Impact of Tokenization on Language Models: An Analysis for Turkish

Tokenization is an important text preprocessing step to prepare input tokens for deep language models. WordPiece and BPE are de facto methods employed by models, such as BERT GPT. However, the impact of tokenization can be different morphologically rich languages, Turkic where many words generated adding prefixes suffixes. We compare five tokenizers at granularity levels, i.e. their outputs vary from smallest pieces characters surface form words, including a Morphological-level tokenizer. train these pretrain medium-sized models using RoBERTa pretraining procedure on Turkish split OSCAR corpus. then fine-tune our six downstream tasks. Our experiments, supported statistical tests, reveal that tokenizer has challenging performance with tokenizers. Furthermore, we find increasing vocabulary size improves Morphological Word-level more than The ratio number parameters total model empirically chosen 20% 40% other obtain reasonable trade-off between performance.